Method for categorizing a scene comprising a sub-scene with machine learning

ABSTRACT

A method for identifying a scene, comprising a computing device receiving a plurality of data points corresponding to a scene; the computing device determining one or more subsets of data points from the plurality of data points that are indicative of at least one sub-scene in said scene, said at least one sub-scene displayed on a display device that is part of said scene, wherein said at least one sub-scene does not represent said scene; the computing device categorizing said scene, disregarding said at least one sub-scene, wherein the categorizing includes interpreting said scene by a computer vision system such that said at least one sub-scene is not taken into account in the categorizing of said scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/160,948 filed Oct. 15, 2018, titled “Method for categorizing a scenecomprising a sub-scene with machine learning,” which is incorporatedherein by reference in its entirety for all purposes.

FIELD OF THE INVENTION

The invention relates to a method, a system, a device, and a computerprogram product for training a machine learning model for categorizing ascene, a subject, a pose or an action in a video, an image, a series ofimages or a live stream.

BACKGROUND OF THE INVENTION

Artificial intelligence (AI) is developing rapidly and AI applicationsare supporting or will support all industries including the aerospaceindustry, agriculture, chemical industry, computer industry,construction industry, defense industry, education industry, energyindustry, entertainment industry, financial services industry, foodindustry, health care industry, hospitality industry, informationindustry, manufacturing, mass media, mining, telecommunication industry,transport industry, water industry and direct selling industry.

Human-machine communication becomes more and more important. Machines(such as computers, smartphones, tablets and robots) are penetratingsociety rapidly.

Computer vision is an area of AI wherein machine learning can be used toclassify scenes in images of living beings and objects having poses andshowing actions.

US20160180189, with title “Image Scene Recognition”, according to itsabstract describes “Systems, methods, and computer program products maybe directed to creating an image hash. Key points can be identified atdifferent locations within a sample image. Descriptor vectors for thekey points can be identified, the descriptor vectors describing localimage information around the key points, where each descriptor vector isan n-dimensional array. Key points can be generated based on hashes ofdata vectors that include at least one of the descriptors, where eachfeature is a 36×20 hash value.”

U.S. Pat. No. 8,189,866, with title “Human-action recognition in imagesand videos”, according to its abstract describes “The present disclosureincludes, among other things, systems, methods and program productsapplying a plurality of low-level feature detectors to an image whereeach low-level feature detector produces a respective low-level featurevector that represents a detection result. The low-level feature vectorsare provided to a plurality of higher-level feature detectors where eachhigher-level feature detector produces a respective higher-level featurevector that represents a detection result based on a distribution offeatures in one or more of the low-level feature vectors. Thehigher-level feature vectors are then provided to a classifier in orderto classify a human-action in the image.”

US20100303303, with title “Methods for recognizing pose and action ofarticulated objects with collection of planes in motion”, according toits abstract describes “The invention comprises an improved system,method, and computer-readable instructions for recognizing pose andaction of articulated objects with collection of planes in motion. Themethod starts with a video sequence and a database of referencesequences corresponding to different known actions. The methodidentifies the sequence from the reference sequences such that thesubject in performs the closest action to that observed. The methodcompares actions by comparing pose transitions. The cross-homographyinvariant may be used for view-invariant recognition of human body posetransition and actions.”

US20090190798, with title “Provided are a system and method forreal-time object recognition and pose estimation using in-situmonitoring. The method includes the steps of: a) receiving 2D and 3Dimage information, extracting evidences from the received 2D and 3Dimage information, recognizing an object by comparing the evidences withmodel, and expressing locations and poses by probabilistic particles; b)probabilistically fusing various locations and poses and finallydetermining a location and a pose by filtering inaccurate information;c) generating ROI by receiving 2D and 3D image information and thelocation and pose from the step b) and collecting and calculatingenvironmental information; d) selecting an evidence or a set ofevidences probabilistically by receiving the information from the stepc) and proposing a cognitive action of a robot for collecting additionalevidence; and e) repeating the steps a) and b) and the steps c) and d)in parallel until a result of object recognition and pose estimation isprobabilistically satisfied.”

US20030058111, with title “Computer vision based elderly care monitoringsystem”, according to its abstract describes “A method for monitoring aperson of interest in a scene, the method comprising: capturing imagedata of the scene; detecting and tracking the person of interest in theimage data; analyzing features of the person of interest; and detectingat least one of an event and behavior associated with the detectedperson of interest based on the features; and informing a third party ofthe at least one detected events and behavior.”

SUMMARY OF THE INVENTION

The invention allows a device, or a system, or a computer programproduct for running on a computing device to interpret a scenecorrectly.

Computer vision is a science that tries to understand what can be seenand what is happening in an image or series of images such as a photopicture, a video or a live stream. To that extend, machine learning canbe used. An image contains a scene reflecting people, animals and/orobjects often executing an activity. Current computer vision technologyfails when a display device in a scene is showing a sub-scene. Forinstance, when computer vision should detect the number of peoplegathered in a hall, the people on various display devices such asbillboards, posters and television screens should not be counted.

Hence, it is an aspect of the invention to provide an improved and/oralternative method for training machine learning models.

To that end, there is provided a method for categorizing an action in ascene, comprising:

a computing device receiving a plurality of data points corresponding tosaid scene;

the computing device determining one or more subsets of data points fromthe plurality of data points, wherein said one or more subsets of datapoints are indicative of at least one sub-scene in said scene, saidsub-scene displayed on a display device;

the computing device categorizing said action in said scene,disregarding said a least one sub-scene.

There is further provided a device comprising an AI system forcategorizing an action in a scene, said AI system comprising a computingdevice running a computer program performing:

receiving a plurality of data points corresponding to said scene;

determining one or more subsets of data points from the plurality ofdata points, wherein said one or more subsets of data points areindicative of at least one sub-scene in said scene, said sub-scenedisplayed on a display device;

categorizing said action in said scene, said computer programdisregarding said a least one sub-scene.

There is further provided a computer program product for running on acomputing device, performing:

receiving a plurality of data points corresponding to said scene;

determining one or more subsets of data points from the plurality ofdata points, wherein said one or more subsets of data points areindicative of at least one sub-scene in said scene, said sub-scenedisplayed on a display device;

categorizing said action in said scene, said computer programdisregarding said a least one sub-scene.

It was found that display devices, such as televisions, (computer)screens, billboards, posters and other displays (such as prints onclothes and other fabrics) might be part of a scene while showing asub-scene. When classifying a scene with computer vision, a sub-scene ona display device can have an impact on the classification of the sceneand the elements within the scene. This impact is often noise whichpreferably is avoided when classifying and interpreting a scene.

The invention allows a device, or a system, or a computer programproduct for running on a computing device to interpret a scenecorrectly. Even when the scene comprises a display device showing asub-scene.

In the current context, categorization take place and the at least onesub-scene is disregarded in the process of categorizing. This may bedone before the actual categorizing is done. In an embodiment, this mayalso be done during, or while, the actual categorizing is done. Thismeans within a functional timeframe during which the actual categorizingis done. In fact, the functional result is such that the sub-scene isnot taken into account in the categorizing.

A subject can be living being, i.e. an animal or a person, or an object.A physical product is an example of an object, as is a car, a statue ora house.

A scene is a view or picture of a place with at least one subject. Ascene can be a view or picture of an event, or activity.

An activity in an embodiment is a series of actions.

An action is a movement of a subject having trajectory.

A pose, as referred to in computer vision, is the position andorientation of a subject. A body of a living being has a pose. Also, avehicle has a pose which can be defined by its position and orientation.The pose of a living being can be detected by articulated body poseestimation.

An image or a series of images or a time series of images can forinstance result from a LIDAR, a visual light camera, a sonar imaging, aradar imaging, a laser imaging, or an infrared camera.

An image capturing device in an embodiment is a device that can providean image or a series of images or a time series of images, in particulara digital image or digital picture. Such a device can comprise a cameraof a filming (motion picture) device. Examples are devices comprising aCCD or similar imaging elements. Other examples of image capturingdevices are a camera, a sonar, a radar, a laser, LIDAR and an infraredcamera. As such, these devices are known to a skilled person.

A computing device is any machine for automatically executingcalculations or instructions. A computing device might be a PC, aserver, a cloud server, a locally distributed server environment, acomputer cloud environment or any circuitry for performing particularfunctions in an electronic device.

A computing device may output a confidence value associated with one ormore of the categories. Also, other means of determining a sub-scene, ifany is being provided, are possible as well.

In some embodiments, once the computing device determines acategorization for the one or more subsets of data points, the computingdevice may store a given label associated with the determined categoryfor the plurality of data points. The plurality of data points may thenbecome part of the training data which may be used for futuredeterminations of scenes and sub-scenes.

A computing device may identify patterns using the machine learningalgorithm to optimize sub-scene detection, and/or scene detection ingeneral. For instance, the machine learning algorithm may indicate thatan average display device in a sub-scene is at least 350 square inches,a respective size of a given display device being a possible featurevector is utilized by the computing device (e.g. by the machine learningalgorithm) to identify the average display device size.

Categorization may involve identifying to which of a set of categories(e.g. normal condition scene or emergency scene and/or allowed action orprohibited action and/or awkward pose or normal pose and/or ordinaryobject or out-of-the-ordinary object) a new captured scene may belong,on the basis of a set of training data with known categories, such asthe aforementioned categories. Categorization of the one or more subsetsof data points associated with a captured scene may be performed usingone or more machine learning algorithms and statistical classificationalgorithms. Example algorithms may include linear classifiers (e.g.Fisher's linear discriminant, logistic regression, naive Bayes, andperceptron), support vector machines (e.g. least squares support vectormachines), clustering algorithms (e.g. k-means clustering), quadraticclassifiers, multi-class classifiers, kernel estimation (e.g. k-nearestneighbor), boosting, decision trees (e.g. random forests), neuralnetworks, Gene Expression Programming, Bayesian networks, hidden Markovmodels, binary classifiers, and learning vector quantization. Otherexample classification algorithms are also possible.

The process of categorization may involve the computing devicedetermining, based on the output of the comparison of the one or moresubsets with the one or more predetermined sets of scene types, aprobability distribution (e.g. a Gaussian distribution) of possiblescene types associated with the one or more subsets. Those skilled inthe art will be aware that such a probability distribution may take theform of a discrete probability distribution, continuous probabilitydistribution, and/or mixed continuous-discrete distributions. Othertypes of probability distributions are possible as well.

In order to detect and localize a subject in a scene from a capturedimage an embodiment uses a method to detect subjects. Such a method willuse machine learning techniques (mainly deep learning) to design andtrain a model which detects subjects given an input of a visualrepresentation, e.g. an RGB image, as the system perceives. The model istrained on a large amount of annotated data; it comprises images withand without subjects and locations of the subjects are annotated.

In the case of deep learning, a detection framework such as Faster-RCNN,SSD, R-FCN, Mask-RCNN, or one of their derivatives can be used. A basemodel structure can be VGG, AlexNet, ResNet, GoogLeNet, adapted from theprevious, or a new one. A model can be initialized with weights andtrained similar tasks to improve and speedup the training Optimizing theweights of a model, in case of deep learning, can be done with the helpof deep learning frameworks such as Tensorflow, Caffe, or MXNET. Totrain a model, optimization methods such as Adam or RMSProb can be used.Classification loss functions such Hinge Loss or Softmax Loss can beused. Other approaches which utilize handcrafted features (such as LBP,SIFT, or HOG) and conventional classification methods (such as SVM orRandom Forest) can be used.

In order to detect and localize a living being in a scene from aretrieved image an embodiment uses a method to detect living beings.Such a method will use machine learning techniques (mainly deeplearning) to design and train a model which detects living beings givenan input of a visual representation, e.g. an RGB image, as the systemperceives. The model is trained on a large amount of annotated data; itcomprises images with and without living beings and locations of theliving beings are annotated.

In the case of deep learning, a detection framework such as Faster-RCNN,SSD, R-FCN, Mask-RCNN, or one of their derivatives can be used. A basemodel structure can be VGG, AlexNet, ResNet, GoogLeNet, adapted from theprevious, or a new one. A model can be initialized with weights andtrained similar tasks to improve and speedup the training. Optimizingthe weights of a model, in case of deep learning, can be done with thehelp of deep learning frameworks such as Tensorflow, Caffe, or MXNET. Totrain a model, optimization methods such as Adam or RMSProb can be used.Classification loss functions such Hinge Loss or Softmax Loss can beused. Other approaches which utilize handcrafted features (such as LBP,SIFT, or HOG) and conventional classification methods (such as SVM orRandom Forest) can be used.

To detect bodily features, the system in an embodiment can determine keypoints on the body (e.g. hands, legs, shoulders, knees, etc.) of aliving being.

To detect the key points on the body of a living being, in an embodimentthe system comprises a model that is designed and trained for thisdetection. The training data to train the model comprises an annotationof various key points locations. When a new image is presented, themodel allows identification of the locations of such key points. To thisend, the system can utilize existing key point detection approaches suchas MaskRCNN or CMU Part Affinity Fields. The training procedure and datacan be customized to best match the context of the content of theretrieved images. Such context may comprise an indoor context (like ahome, a shop, an office, a station, an airport, a hospital, a theatre, acinema etc.) or an outdoor context (like a beach, a field, a street, apark etc.) wherein there are changing lighting conditions.

For example, a pretrained deep neural network (DNN) on ImageNet, e.g.VGGNet, AlexNet, ResNet, Inception and Xception, can be adapted bytaking the convolution layers from these pretrained DNN networks, and ontop of them adding new layers specially designed for scene recognitioncomprising one or more display devices, and train the network asdescribed for the model. Additional new layers could comprise speciallydesigned layers for action and pose recognition. All the aforementionedlayers (scene recognition, pose and action recognition) can be trainedindependently (along with/without the pre-trained conventional layers)or trained jointly in a multi-task fashion.

In an embodiment, the scene is an indoor scene.

In an embodiment, the scene is an outdoor scene.

In an embodiment, the scene comprises a series of subsequent scenesdefining said action.

In an embodiment, the action comprises a traffic action comprising atleast one scene from a viewpoint inside a vehicle looking out of saidvehicle.

There is provided an AI system comprising a computing device running thecomputer program product.

There is further provided an apparatus comprising the AI system, whereinsaid scene comprises a representation of a surrounding of said apparatuscomprising said action, said AI system providing instructions to adjustat least one physical parameter of said apparatus based upon saidcategorizing of said action. Such a physical parameter comprises one ofspeed, direction, pose, position, and orientation in a space.

In an embodiment, such an apparatus comprises an image capturing devicedescribed above.

In an embodiment of the apparatus, it is selected from a vehicle and arobot system. Examples of a vehicle are an automobile, a cart, a train,a ship, a vessel, a submarine, an airplane, and a helicopter. Examplesof a robot system comprise a drone, a robotic arm, a humanoid andautonomously operating constructions.

There is further provided a monitoring system comprising the AI system,wherein said scene comprises a representation of a surrounding of saidmonitoring system comprising said action, said AI system providing asignal based upon said categorizing of said action. In an embodiment, asignal comprises a notification, a control signal, a text message, anelectromagnetic signal and an optical signal.

In an embodiment, such a monitoring system comprises an image capturingdevice.

In an embodiment, there is provided a surveillance system comprising themonitoring system described above.

The invention further relates to a method for categorizing a scene,comprising: a computing device receiving a plurality of data pointscorresponding to said scene; the computing device determining one ormore subsets of data points from the plurality of data points, whereinsaid one or more subsets of data points are indicative of at least onesub-scene in said scene, said sub-scene displayed on a display device;the computing device categorizing said scene, disregarding said a leastone sub-scene.

The invention further relates to a method for categorizing one or moresubjects in a scene, comprising:

a computing device receiving a plurality of data points corresponding tosaid scene;

the computing device determining one or more subsets of data points fromthe plurality of data points, wherein said one or more subsets of datapoints are indicative of at least one sub-scene in said scene, saidsub-scene displayed on a display device;

the computing device categorizing said one or more subjects in saidscene, the computing device disregarding said a least one sub-scene.

The invention further relates to a method for categorizing a pose in ascene, comprising:

a computing device receiving a plurality of data points corresponding tosaid scene;

the computing device determining one or more subsets of data points fromthe plurality of data points, wherein said one or more subsets of datapoints are indicative of at least one sub-scene in said scene, saidsub-scene displayed on a display device;

the computing device categorizing said pose in said scene, the computingdevice disregarding said a least one sub-scene.

The embodiments described earlier can be combined with any of theaforementioned methods described.

The term “statistically” when used herein, relates to dealing with thecollection, analysis, interpretation, presentation, and organization ofdata. The analysis may be presented into visual formats like graphs, orother known graphical representations and/or tables.

The term “near real-time” or “nearly real-time” (NRT), intelecommunications and computing, refers to the time delay introduced,by automated data processing or network transmission, between theoccurrence of an event and the use of the processed data, such as fordisplay or feedback and control purposes. For example, a near-real-timedisplay depicts an event or situation as it existed at the current timeminus the processing time, as nearly the time of the live event.

The distinction between the terms “near real time” and “real time” issomewhat nebulous and must be defined for the situation at hand. Theterm implies that there are no significant delays. In many cases,processing described as “real-time” would be more accurately describedas “near real-time”. In fact, this may also be described as“functionally real-time”.

Near real-time also refers to delayed real-time transmission of voiceand video. It allows playing video images, in approximately real-time,without having to wait for an entire large video file to download.Incompatible databases can export/import to common flat files that theother database can import/export on a scheduled basis so that they cansync/share common data in “near real-time” with each other.

Real-time signal processing is necessary, but not sufficient in and ofitself, for live signal processing such as what is required in liveevent support. Live audio digital signal processing requires bothreal-time operation and a sufficient limit to throughput delay so as tobe tolerable to performers using stage monitors or in-ear monitors andnot noticeable as lip sync error by the audience also directly watchingthe performers. Tolerable limits to latency for live, real-timeprocessing is a subject of investigation and debate but is estimated tobe between 6 and 20 milliseconds.

A real-time system has been described in Wikipedia as one which“controls an environment by receiving data, processing them, andreturning the results sufficiently quickly to affect the environment atthat time”. The term “real-time” is also used in simulation to mean thatthe simulation's clock runs at the same speed as a real clock, and inprocess control and enterprise systems to mean “without significantdelay”.

The distinction between “near real-time” and “real-time” varies, and thedelay is dependent on the type and speed of the transmission. The delayin near real-time is typically of the order of several seconds toseveral minutes.

Often, systems that are described or seen as “real-time” arefunctionally real-time.

Demography in general is the statistical study of populations,especially human beings (see Wikipedia). As a very general science, itrelates to analysing any kind of dynamic living population, i.e., onethat changes over time or space. Demography encompasses the study of thesize, structure, and distribution of these populations, and spatial ortemporal changes in them in response to birth, migration, aging, anddeath.

Demographic analysis can cover whole societies or groups defined bycriteria such as education, nationality, religion, and ethnicity.

Formal demography limits its object of study to the measurement ofpopulation processes, while the broader field of social demography orpopulation studies also analyses the relationships between economic,social, cultural, and biological processes influencing a population.

The common variables that are gathered in demographic research includeage, sex, income level, race, employment, marital state, occupation,religion, location, home ownership and level of education. Demographicsmake certain generalizations about groups to identify customers.Additional demographic factors include gathering data on preferences,hobbies, lifestyle and more.

A camera is defined in for instance Wikipedia as an optical instrumentfor recording or capturing images, which may be stored locally,transmitted to another location, or both. The images may be individualstill photographs or sequences of images constituting videos or movies.The camera is a remote sensing device as it senses subjects without anycontact. Current cameras are in general digital image recording devices.A camera in general works with the light of the visible spectrum or withother portions of the electromagnetic spectrum. A still camera is anoptical device which creates a single image of an object or scene andrecords it on an electronic sensor. A movie camera or a video cameraoperates similarly to a still camera, except it records a series ofstatic images in rapid succession, commonly at a rate of 24 frames persecond.

The term “substantially” herein, such as in “substantially all emission”or in “substantially consists”, will be understood by the person skilledin the art. The term “substantially” may also include embodiments with“entirely”, “completely”, “all”, etc. Hence, in embodiments theadjective substantially may also be removed. Where applicable, the term“substantially” may also relate to 90% or higher, such as 95% or higher,especially 99% or higher, even more especially 99.5% or higher,including 100%. The term “comprise” includes also embodiments whereinthe term “comprises” means “consists of”.

The term “functionally” will be understood by, and be clear to, a personskilled in the art. The term “substantially” as well as “functionally”may also include embodiments with “entirely”, “completely”, “all”, etc.Hence, in embodiments the adjective functionally may also be removed.When used, for instance in “functionally parallel”, a skilled personwill understand that the adjective “functionally” includes the termsubstantially as explained above. Functionally in particular is to beunderstood to include a configuration of features that allows thesefeatures to function as if the adjective “functionally” was not present.The term “functionally” is intended to cover variations in the featureto which it refers, and which variations are such that in the functionaluse of the feature, possibly in combination with other features itrelates to in the invention, that combination of features is able tooperate or function. For instance, if an antenna is functionally coupledor functionally connected to a communication device, receivedelectromagnetic signals that are receives by the antenna can be used bythe communication device. The word “functionally” as for instance usedin “functionally parallel” is used to cover exactly parallel, but alsothe embodiments that are covered by the word “substantially” explainedabove. For instance, “functionally parallel” relates to embodiments thatin operation function as if the parts are for instance parallel. Thiscovers embodiments for which it is clear to a skilled person that itoperates within its intended field of use as if it were parallel.

Furthermore, the terms first, second, third and the like in thedescription and in the claims, are used for distinguishing betweensimilar elements and not necessarily for describing a sequential orchronological order. It is to be understood that the terms so used areinterchangeable under appropriate circumstances and that the embodimentsof the invention described herein are capable of operation in othersequences than described or illustrated herein.

The devices or apparatus herein are amongst others described duringoperation. As will be clear to the person skilled in the art, theinvention is not limited to methods of operation or devices inoperation.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims. In the claims, any reference signsplaced between parentheses shall not be construed as limiting the claim.Use of the verb “to comprise” and its conjugations does not exclude thepresence of elements or steps other than those stated in a claim. Thearticle “a” or “an” preceding an element does not exclude the presenceof a plurality of such elements. The invention may be implemented bymeans of hardware comprising several distinct elements, and by means ofa suitably programmed computer. In the device or apparatus claimsenumerating several means, several of these means may be embodied by oneand the same item of hardware. The mere fact that certain measures arerecited in mutually different dependent claims does not indicate that acombination of these measures cannot be used to advantage.

The invention further applies to an apparatus or device comprising oneor more of the characterizing features described in the descriptionand/or shown in the attached drawings. The invention further pertains toa method or process comprising one or more of the characterizingfeatures described in the description and/or shown in the attacheddrawings.

The various aspects discussed in this patent can be combined in order toprovide additional advantages. Furthermore, some of the features canform the basis for one or more divisional applications.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of exampleonly, with reference to the accompanying schematic drawings in whichcorresponding reference symbols indicate corresponding parts, and inwhich:

FIGS. 1A-D schematically depict flow charts of example methods tocategorize various aspects from a scene comprising a display device.

FIG. 2 schematically depicts an embodiment for monitoring the indoors ofan elderly home with a television display.

FIG. 3 schematically depicts an embodiment for monitoring a square withdisplay devices.

FIG. 4 schematically depicts an embodiment built into a self-driving caranalyzing a scene comprising a billboard.

The drawings are not necessarily to scale.

DESCRIPTION OF PREFERRED EMBODIMENTS

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols identify similarcomponents, unless context dictates otherwise.

FIGS. 1A-D schematically depict flow charts of example methods (1, 1′,1″ and 1′″) for categorizing wherein a computing device 3 receives datapoints (100, 100′) from scene 10, comprising a display device 2 and asub-scene 10′, resulting is a categorized action 21, a categorized scene20 and one or more categorized subjects 22.

In FIG. 1A, method 1 categorizes an action in scene 10 resulting in acategorized action 21. Computing device 3 receives a plurality of datapoints 100 corresponding to scene 10. Computing device 3 determines asubset of data points 100′ indicative of sub-scene 10′ displayed ondisplay device 2, and categorizes an action in scene 10 and disregardssub-scene 10′.

In FIG. 1B, method 1′ categorizes a scene 10 resulting in a categorizedscene 20. Computing device 3 receives a plurality of data points 100corresponding to scene 10. Computing device 3 determines a subset ofdata points 100′ indicative of sub-scene 10′ displayed on display device2, and categorizes scene 10 and disregards sub-scene 10′.

In FIG. 1C, method 1″ categorizes one or more subjects in scene 10resulting in one or more categorized subjects 22. Computing device 3receives a plurality of data points 100 corresponding to scene 10.Computing device 3 determines a subset of data points 100′ indicative ofsub-scene 10′ displayed on display device 2, and categorizes one or moresubjects in scene 10 and disregards sub-scene 10′.

In FIG. 1D, method 1′″ categorizes a scene 10 resulting in a categorizedscene 20 and a categorized action 21. Computing device 3 receives aplurality of data points 100 corresponding to scene 10. Computing device3 determines a subset of data points 100′ indicative of sub-scene 10′displayed on display device 2, and categorizes scene 10 whiledisregarding sub-scene 10′, wherein a categorized action 21 is deductedfrom a categorized scene 20.

In another method a categorized scene 20 is deducted from one or morecategorized actions (21). For example, a box match scene with variousbillboards can be categorized directly or can be categorized by theactivity or series of actions by boxers fighting in a ring.

In yet another method a categorized scene 20 is deducted from one ormore categorized subjects (22). For example, a box match scene withvarious billboards can be categorized directly or can be categorized bya one or more subjects such as a boxing ring, boxers, trainers, crowdand various attributes in scene 10.

The methods (1, 1′, 1″ and 1′″) may include one or more operations,functions, or actions as depicted in FIGS. 1A-D and may result in one ormore categorized objects as depicted by the blocks 20, 21 and 22.Although the blocks 20, 21 and 22 are depicted in a sequential order,these blocks may in some instances be performed in parallel, and/or in adifferent order than those described herein. Also, the various blocksmay be combined into fewer blocks, divided into additional blocks,and/or removed based upon the desired implementation.

In addition, for the methods (1, 1′, 1″ and 1′″) and other processes andmethods disclosed herein, the flow charts show functionality andoperation of possible implementations of embodiments. In this regard,each method may represent a module, a segment, or a portion of programcode, which includes one or more instructions executable by a processorfor implementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ormemory, for example, such as a storage device including a disk or harddrive. The computer readable medium may include a non-transitorycomputer readable medium, for example, such as computer-readable mediathat stores data for short periods of time like register memory,processor cache and random-access memory (RAM). The computer readablemedium may also include non-transitory media or memory, such assecondary or persistent long-term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer readable media may also be any other volatile ornon-volatile storage systems. The computer readable medium may beconsidered a computer readable storage medium, a tangible storagedevice, or other article of manufacture, for example.

In addition, for the methods (1, 1′, 1″ and 1′″) and other processes andmethods disclosed herein, computing device 3 may represent circuitrythat is wired to perform the specific logical functions in the process.For the sake of example, the methods (1, 1′, 1″ and 1′″) shown in FIGS.1A-D will be described as implemented by an example computing device,such as the computing device 3 depicted in FIG. 2 . The methods (1, 1′,1″ and 1′″) can also be described as implemented by an autonomousvehicle, as depicted in FIG. 4 , as the computing device may be onboardthe vehicle or may be off-board but in wireless communication with thevehicle. It should be understood that other entities or combinations ofentities can implement one or more steps of the example methods (1, 1′,1″ and 1″).

FIG. 2 schematically depicts an application of an embodiment formonitoring an indoors scene of an elderly home 50. In the indoors sceneof elderly home 50 there is a television, as display device 2. A videocamera, as image capturing device 4, captures scene 10 and transmits itsdata points 100 including sub-scene 10′ with data points 100′ tocomputing device 3, operationally coupled to video camera 4. Althoughscene 10 comprises a man with a gun 21, computing device 3 does notcategorizes scene 10 in FIG. 2 as threatening or as a crime scene sincecomputing device 3, when categorizing the scene, is trained to disregardthe data points 100′ of sub-scene 10′ displayed on the television 2.

In another application computing device is categorizing, within scene10, an action, a pose, a subject or a combination thereof.

FIG. 3 schematically depicts an application of an embodiment formonitoring an outdoors scene of a square 51. In the outdoors scene ofsquare 51 there is a wide screen, as display device 2 and a merchandisewagon 7 with a display device 2′. A video camera, as image capturingdevice 4, monitors the square and is operationally coupled to computingdevice 3. Similar to the indoors example of FIG. 2 , the monitoring ofan outdoors scene of square 51 is complicated by the display devices (2and 2′). Wide screen 2 is showing a fighting scene 22 while the displaydevice 2′ on the merchandise wagon 7 shows similar architecture as thesurroundings of the square.

In this application, computing device 3, when categorizing the people 8on the square, will, by disregarding the sub-scene on wide screen 2,deduct that the number of people on the square in view of camera 4 isnine. For instance, such information can be used for monitoring andcontrolling a crowd in an open space.

Additional in this application, computing device 3, when categorizingthe houses 9 on the square, will, by disregarding the sub-scene ondisplay device 2′, deduct that the number of houses in view of camera 4is three and by doing so, it will also increase the correctcategorization for merchandise wagon 7 since computing device 3 is notmisled by the display device 2′.

Would this application serve as a surveillance system then the system ofFIG. 3 would not trigger an alarm for an alleged fighting incident 22 asdisplayed on the wide screen (display device 2). Adjacent, thesurveillance system could trigger an alarm for merchandise wagon 7 fortrespassing the square by an unauthorized vehicle.

FIG. 4 schematically depicts an embodiment built into a self-driving car5 analyzing a scene 10 comprising a billboard, as display device 2.Computing device 3 is operationally coupled with image capturing device4 and receives a plurality of data points 100 corresponding to scene 10.Computing device 3 determines a subset of data points 100′ indicative ofsub-scene 10′ displayed on billboard 2, categorizes scene 10 anddisregards sub-scene 10′. As a result the self-driving car 5 will not bemisled by the image of car 23 on billboard 2.

The billboard 2 can be a traditional poster, a digital billboard or ascreen configured to display a static image, a (time) series of images,or a video movie.

Further, an example system may take the form of a non-transitorycomputer-readable medium, which has program instructions stored thereonthat are executable by at least one processor to provide thefunctionality described herein.

An example system may take the form of any vehicle or a subsystem of anyvehicle that includes such a non-transitory computer-readable mediumhaving such program instructions stored thereon. Therefore, the terms“computing device” and “autonomous vehicle” can be interchangeableherein. However, in some examples, the computing device may beconfigured to control the vehicle in an autonomous or semi-autonomousoperation mode.

In yet another application, an embodiment is built into a robot so therobot will correctly interpreter its surrounding and the scene whereinthe robot is operating.

It may be readily understood that certain aspects of the disclosedsystems and methods can be arranged and combined in a wide variety ofdifferent configurations, all of which are contemplated herein.

It will also be clear that the above description and drawings areincluded to illustrate some embodiments of the invention, and not tolimit the scope of protection. Starting from this disclosure, many moreembodiments will be evident to a skilled person. These embodiments arewithin the scope of protection and the essence of this invention and areobvious combinations of prior art techniques and the disclosure of thispatent.

The invention claimed is:
 1. A method for categorizing a scene,comprising: a computing device receiving a plurality of data pointscorresponding to said scene; the computing device determining one ormore subsets of data points from the plurality of data points, whereinsaid one or more subsets of data points are indicative of at least onesub-scene in said scene, said at least one sub-scene displayed on adisplay device that is part of said scene, wherein said at least onesub-scene does not represent said scene; the computing devicecategorizing said scene, disregarding said at least one sub-scene,wherein the categorizing includes interpreting said scene by a computervision system such that said at least one sub-scene is not taken intoaccount in the categorizing of said scene.
 2. The method of claim 1,wherein said scene is an indoor scene.
 3. The method of claim 1, whereinsaid scene is an outdoor scene.
 4. The method of claim 1, wherein saidscene comprises a series of subsequent scenes defining said scene. 5.The method of claim 1, wherein said scene comprises a traffic scene froma viewpoint inside a vehicle looking out of said vehicle.
 6. A devicecomprising an AI system for categorizing a scene, said AI systemcomprising a computing device running a computer program performing:receiving a plurality of data points corresponding to said scene;determining one or more subsets of data points from the plurality ofdata points, wherein said one or more subsets of data points areindicative of at least one sub-scene in said scene, said at least onesub-scene displayed on a display device that is part of said scene,wherein said at least one sub-scene does not represent said scene;categorizing said scene, said computer program disregarding said atleast one sub-scene, wherein the categorizing includes interpreting saidscene by a computer vision system such that said at least one sub-sceneis not taken into account in the categorizing of said scene.
 7. Anon-transitory computer readable medium having stored thereon computerprogram instructions that, when executed by a processor in a computingdevice, configure the computing device to perform: receiving a pluralityof data points corresponding to a scene; determining one or more subsetsof data points from the plurality of data points, wherein said one ormore subsets of data points are indicative of at least one sub-scene insaid scene, said at least one sub-scene displayed on a display devicethat is part of said scene, wherein said at least one sub-scene does notrepresent said scene; categorizing said scene, said computer programinstructions disregarding said at least one sub-scene, wherein thecategorizing includes interpreting said scene by a computer visionsystem such that said at least one sub-scene is not taken into accountin the categorizing of said scene.
 8. An AI system comprising acomputing device executing the computer program instructions of claim 7.9. An apparatus comprising the AI system of claim 8, wherein said scenecomprises a representation of a surrounding of said apparatus comprisingsaid scene, said AI system providing instructions to adjust at least onephysical parameter of said apparatus based upon said categorizing ofsaid scene.
 10. The apparatus of claim 9, selected from a vehicle and arobot system.
 11. A monitoring system comprising the AI system of claim8, wherein said scene comprises a representation of a surrounding ofsaid monitoring system, said AI system providing a signal based uponsaid categorizing of said scene.
 12. A surveillance system comprisingthe monitoring system of claim 11.